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103, specifying the information indicating the covered background area information 
and the information indicating the uncovered background area information, is fed to 
the separating unit 251, whilst the area information indicating the foreground 
information and the area information indicating the background area are routed to the 

switches 252, 254, respectively. 

The mixing ratio a supplied from the mixing ratio calculating unit 104 is sent 

to the separating unit 25 1. 

The separating unit 251 separates the foreground component from the input 
picture, based on the area information indicating the covered background area 
information, the area information indicating the uncovered background area 
information and the mixing ratio a, and supplies the so-separated foreground 
component to a synthesis unit 253, while separating the background component from 
the input picture to route the so-separated background componentto the synthesis unit 
255. 

When a pixel corresponding to the foreground is input, the switch 252 is closed, 
based on the area information indicating the foreground area, to route only the pixels 
corresponding to the foreground contained in the input picture to the synthesis unit 
253. 

When a pixel corresponding to the background is input, the switch 254 is 
closed, based on me area information indicating me background area, to rou^ 
pixel corresponding to the background contained in the input picture to the synthesis 
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unit 255. 



The synthesis unit 253 synthesizes a foreground component picture, based on 
the component from the separating unit 25 1 , corresponding to the foreground, and on 
the pixel from the switch 252, corresponding to the foreground, to output the 
synthesized foreground component picture. Since the foreground area and the mixed 
area are not overlapped, the synthesis unit 253 applies the processing of logical sum 
to the component corresponding to the foreground and to the pixel corresponding to 
the foreground to synthesize the foreground component picture. 

In the initializing processing, executed first in the processing for synthesizing 
the foreground component picture, the synthesis unit 25 3 stores a picture with all zero 
pixel values in an internal frame memory to store (overwrite) the foreground 
component picture in the processing for synthesizing the foreground component 
picture. Thus, in the pixels corresponding to the background area, in the foreground 
component picture output by the synthesis unit 253, 0s are stored as pixel values. 

The synthesis unit 255 synthesizes the background component picture, based 
on the components fromthe separating unit 25 1 and on the pixels from the switch 254 
corresponding to the background, to output the synthesized background component 
picture. Since the background area and the mixed area are not overlapped, the 
synthesis unit 255 applies the processing of logical sum to the component 
corresponding to the background and to the pixel corresponding to die background to 
synthesize the background component picture. 
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IB the initializing processing, executed first in the processing for synthesizing 
thebackgroundcomponentpicturcthe synthesis unit 255 stores apicture with all zero 
pixel values in an internal frame memory to store (overwrite) the background 
component picture in the processing for synthesizing the background component 
picture. Thus, in the pixels corresponding to the foreground area, in the background 
component picture output by the synthesis unit 255, 0s are stored as pixel values. 

Fig.49 shows an input picture, fed to the foreground/background separating unit 
105, and the foreground component picture and the background component picture, 
output from the foreground/background separating unit 105. 

Fig.49A schematically shows the displayed picture, while Fig.49B 
diagrammatically shows a model obtained on developing one-line pixels comprised of 
pixels belonging to the foreground area, pixels belongingto the background area and 

the pixels in the mixed area, along the time axis. 

ReferrmgtoFigs.49Aand49B,mebackgroundcomponentpicture,outputfrom 

the foregroundVbackground separating unit 105, is comprised of a background 
component contained in the pixels belonging to the background area and pixels 

belonging to the mixed area. 

RefeirmgtoFigs.49Aand49B,theforegroundcomponentpicture,outputfrom 

the foregrounoTbackground separating unit 105, is comprised of a foreground 
component contained in the pixels belonging to the foreground area and pixels 
belonging to the mixed area. 
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The pixel values of the pixels of the mixed area are separated by the 
foreground/background separating unit 105 into the background component and the 
foreground' component. The background component, thus separated, makes up the 
background component picture along with the pixels belonging to the background area. 
The foreground component separated makes up the foreground component picture 
along with the pixels belonging to the foreground area. 

In this manner, the pixel values of the pixels of the foreground component 
picture, associated with the background area, are set to 0, while meaningful pixel 
values are set in the pixels corresponding to the foreground area and to the pixels 
corresponding to the mixed area. Similarly, the pixel values of the pixels of the 
background component picture, associated with the foreground area, are setto 0, while 
meaningful pixel values are set in the pixels corresponding to the background area and 
to the pixels corresponding to the mixed area. 

The processing executed by the separatingunit 25 1 in separating the foreground 
and background components from the pixels belonging to the mixed area is 
explained. 

Fig.50 diagrammatically shows a model of a picture showing foreground and 
background components of two frames including the foreground corresponding to an 
object moving from left to right in the drawing. In the picture model of Fig.50, the 
movement quantity v of the foreground is 4, with the number of times of the virtual 
splitting being 4. 
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In the frame #n, the leftmost pixel and fourteenth to eighteenth pixels from left 
are composed only of the background components and belong to the background area. 
In the frame #n, the second to fourth pixels from left include background and 
foreground components and belong to the uncovered background area. In the frame 
#n, the eleventh to thirteenth pixels from left include background and foreground 
components and belong to the covered background area. In the frame #n, the fifth to 
tenth pixels from left include only foreground components and belong to the covered 
foreground area. 

In the frame #n+l, the first to fifth pixels and the eighteenth pixel from left 
are composed only of the background component and belong to the background area. 
In the frame #n+l, the sixth to eighth pixels from left include background and 
foreground components and belong to the uncovered background area. In the frame 
#n+l, the fifteenth to seventeenth pixels from left include background andforeground 
components and belong to the covered background area. In the frame #n+ 1 , the ninth 
to fourteenth pixels from left include only foreground components and belong to the 
covered foreground area. 

Fig.5 1 illustrates the processing for separating the foreground component from 
the pixels belonging to the covered background area. In Fig.5 1 , «1 to «18 represent 
the mixing ratio values, associated with respective pixels of the frame #n. In Fig.5 1, 
the fifteenth to seventeenth pixels from left belong to the covered background area. 

The pixel value C15 which is the fifteenth pixel from left of the frame #n is 
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represented by the following equation (29): 
CI 5 = B15/v + F09/v + F08/v + F07/v 

= al5-B15 + F09/v + F08/v +■ F07/v 
= al5-P15 + F09/v + F08/v + F07/v 

- (29) 

where al 5 is the mixing ratio of the fifteenth pixel from left of the frame #n and P15 
is the pixel value of the fifteenth pixel from left of the frame #n- 1 . 

Based on the equation (29), the sum fl 5 of the foreground components of the 
fifteenth pixel of the frame #n is represented by the equation (30): 

fl 5 = F09/v + F08/v + F07/v 
= C15-al5-P15 

- (30). 

Similarly, the sum fl6 of the foreground components of the sixteenth pixel of 
the frame #n and the sum first wiring pattern 17 of the foreground components of the 
seventeenth pixel of the frame #n are represented by the equations (3 1) and (32): 

fl6 = C16-al6-P16 

■••(31) 

and 

fl7 = C17-CCl7-P17 

- (32) 

respectively. 
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In this manner, the foreground components fc, contained in the pixel value C 
of the pixel belonging to the covered background area, may be calculated by the 
equation (33): 

fc = CdP 

- (33). 

Fig. 52 illustrates the processing for separatingthe foreground components from 
thepixelsbelongingtotheuncoveredbackgroundarea. In Fig.52,al to al 8 represent 
the values of the rnixing ratio for respective pixels of the frame #n. In Fig.52, the 
second to fourth pixels from left belong to the uncovered background area. 

The pixel values C02 of the second pixel from left of the frame #n are 
represented by the equation (34): 

C02 = B02/v + B02/v + B02/v + F0 1/v 
= CS2-B02 + FOl/v 
= a2-N02 + FOl/v 

••• (34) 

where Ct2 is a mixing ratio of the second pixel from left of the frame #n and N02 is a 
pixel value of the second pixel from left of the frame #n+l. 

Based on the equation (34), the sum f02 of the foreground component of the 
second pixel from left of the frame #n is represented by the equation (35): 

f02 = FOl/v 

= C02-CC2-NO2 
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- (35) 

Similarly, the sum f03 of the foreground components of the third pixel from left 
of the frame #n and the sum f04 of the foreground components of the fourth pixel 
from left of the frame #n are represented by the following equations (36) and (37): 

f03 = C03-Ct3-NO3 

-(36) 

and 

f04 = C04-CC4-NO4 

•••(37) 

respectively. 

The foreground component fu included in the pixel value C of the pixel 
belonging to the uncovered background area is calculated by the following equation 
(38): 

fu = C*a-N 

-(38) 

where N is a pixel value of a corresponding pixel of the next frame. 

In this manner, the separating unit 251 is able to separate the foreground 
components and the background components from the pixels belonging to the mixed 
area, based on the information indicating the covered background area and on the 
mformationmdicatmgmecoveredbackgroundar e a,contamedmme 

and on me pixel-based mixing ratio a. 
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In Fig. 5 3, which is a block diagram showing an illustrative structure of the 
separating unit 251 adapted for executing the above-described processing, a picture 
input to the separating unit 251 is input to the frame memory 301, whilst the area 
information indicating the covered background area and the uncovered background 
area, and the mixing ratio a, are input to a separating processing block 302. 

The frame memory 301 stores the input pictures on the frame basis. If the 
object of processing is the frame #n, the frame memory 301 stores the frame #n'l, 
directly previous to the frame #n, frame #n and the frame #n+l next to the frame #n. 

The frame memory 301 routes the corresponding pixels of the frame #n*l, 
frame #n and the frame #n+l, to a separating processing block 302. 

Based on the area information indicating the covered background area 
information and the uncovered background area information, and on the mixing ratio 
a, the separating processing block 302 performs calculations, explained with reference 
to Figs.51 and 52, on the pixel values of the corresponding pixels of the frame #n l, 
frame #n and the frame #n+l, supplied from the frame memory 301, to separate the 
foreground and background components from the pixels belonging to the mixed area 
of the frame #n to route the separated components to a frame memory 303. 

The separating processing block 302 is made up of an uncovered area processor 
3 1 1 , a covered area processor 3 12, a synthesis unit 3 13 and a synthesis unit 3 14. 

The uncovered area processor 311 includes a multiplier 32 1 which multiplies 
the pixel value of the pixel of the frame #n+l supplied from the frame memory 301 
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with the mixing ratio CL to route the resulting product to the switch 322, which is 
closed when the pixel of the frame #n (corresponding to the pixel of the frame #n-H) 
supplied from the frame memory 3 0 1 is in the uncovered background area, to route the 
pixel value multiplied with the mixing ratio a sent from the multiplier 321 to an 
operating unit 322 and to the synthesis unit 3 14. The pixel value of the pixel of the 
frame #n+l from the switch 322, multiplied by the mixing ratio CC, is equal to the 
background component of the pixel value of the corresponding pixel of the frame #n.. 

An operating unit 323 subtracts the background component supplied from the 
switch 322 from the pixel value of the pixel of the frame #n supplied from the frame 
memory 301 to find the foreground component. The operating unit 323 routes the 
foreground component of the pixel of the frame #n, belonging to the uncovered 
background area, to the synthesis unit 313. 

The covered area processor 312 includes a multiplier 331 which multiplies the 
pixel value of the pixel of the frame #n* 1 supplied from the frame memory 301 with 
the mixing ratio a to route the resulting product to the switch 322, which is closed 
when the pixel of the frame #n (corresponding to the pixel of the frame #n- 1) supplied 
from the frame memory 30 1 is in the covered background area, to route the pixel value 
multiplied with the mixing ratio 0t sent from the multiplier 33 1 to an operating unit 333 
and to the synthesis unit 314. The pixel value of the pixel of the frame #n* 1 from the 
switch 332, multiplied by the mixing ratio a, is equal to the background component 
of the pixel value of the corresponding pixel of the frame #n. 
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An operating unit 333 subtracts the background component supplied from the 
switch 332 from the pixel value of the pixel of the frame #n supplied from the frame 
memory 301 to find the foreground component. The operating unit 333 routes the 
foreground component of the pixel of the frame #n, belonging to the covered 
background area^ to the synthesis unit 313. 

The synthesis unit 3 13 synthesizes the foreground component of the pixel from 
the operating unit 323, belonging to the uncovered background area, to the foreground 
component of the pixel belonging to the covered background area, to route the 
resulting sum to the frame memory 303. 

The synthesis unit 314 synthesizes the background component of the pixel 
from the operating unit 323, belonging to the uncovered background area, to the 
background component of the pixel from the switch 332 belonging to the covered 
background area, to route the resulting sum to the frame memory 303. 

The frame memory 303 stores the foreground and background components of 
the pixels of the mixed area of the frame #n» supplied from the separating processing 
block 302. 

The frame memory 303 outputs the stored foreground component of the pixels 
of the mixed area of the frame #n and the stored background component of the pixels 
of the mixed area of the frame #n. 

By exploiting the mixing ratio a as a characteristic value, the foreground and 
background components contained in the pixel value can be separated completely from 
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each other. 

The synthesis unit 253 synthesizes the foreground component of the pixels of 
the mixed area of the frame #n , output by the separating unit 251, and the pixels 
belonging to the foreground area, to each other, to generate a foreground component 
picture. The synthesis unit 255 synthesizes the background component of the pixels 
of the mixed area of the frame #n , output by the separating unit 25 1, and the pixels 
belonging to the background area, to each other, to generate a background component 
picture. 

Fig. 54 shows an example of a foreground component picture and an example 
of the background component picture corresponding to the frame #n of Fig.50. 

Fig. 54 A shows an example of the foreground component picture corresponding 
to the frame #n of Fig.50. The pixel values of the leftmost pixel and the fourteenth 
pixel from left were composed only of the background component before foreground/ 
background separation, and hence are equal to 0. 

The second to fourth pixels from left belonged to the uncovered background 
area before foreground/ background separation, with the background component being 
0 and with the foreground component being left intact. The eleventh to thirteenth 
pixels from left belonged to the covered background area before foreground/ 
background separation, with the background component being 0 and with the 
foreground component being left intact. The fifth to tenth pixels from left are left 
intact because these pixels are composed only of the background components. 
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Fig.54B shows an example of a background component picture corresponding 
to the frame #n of Fig. 50. The leftmost pixel and the fourteenth pixel from left are 
composed only of the background component before foreground/ background 
separation, and hence are left intact. 

The second to fourth pixels from left belonged to the uncovered background 
area before foreground/ background separation, with the foreground component being 
0 and with the background component being left intact. The eleventh to thirteenth 
pixels from left belonged to the covered background area before 
foreground/background separation, with the background component being 0 and with 
the foreground component being left intact. The pixel values of the fifth to tenth 
pixels from left are set to zero because these pixels are composed only of the 
foreground components. 

Referring to the flowchart of Fig.55, the processing for foreground/background 
separation by the foreground/background separating unit 105 is explained. At step 
S201, the frame memory 301 of the separating unit 251 acquires an input picture and 
stores the frame #n, to be processed for foreground/background separation, along with 
the previous frame #n- 1 and the subsequent frame #n+l . 

At step S202, the separating processing block 302 of the separating unit 251 
acquires the area information supplied from the mixing ratio calculating unit 104. At 
step S203, the separating processing block 302 of the separating unit 251 acquires the 
mixing ratio a routed from the mixing ratio calculating unit 104. 
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At step S204, the uncovered area processor 311 extracts the background 
component, based on the area information and the mixing ratio a, the pixel values of 
pixels belonging to the uncovered background area, supplied from the frame memory 



301. 



At step S205, the uncovered area processor 311 extracts the foreground 
component, based on the area information and the mixing ratio a, the pixel values of 
pixels belonging to the uncovered background area, supplied from the frame memory 
301. 

At step S206, the covered area processor 312 extracts the background 
component, based on the area information and the mixing ratio «, the pixel values of 
pixels belonging to the covered background area, supplied from the frame memory 



301. 



At step S207, the covered area processor 312 extracts the foreground 
component, based on the area information and the mixing ratio a, the pixel values of 
pixels belonging to the covered background area, supplied from the frame memory 
301. 

At step S208, the synthesis unit 313 synthesizes the foreground component, 
belonging to the uncovered background area, extracted by the processing of step S205, 
and the foreground component, belonging to the covered background area, extracted 
by the processing of step S207. The synthesized foreground component is routed to 
the synthesis unit 253, which then synthesizes the pixels belonging to the foreground 



PAGE 15/51 * RCVD AT 7/16/2009 1:11:05 PM [Eastern Daylight Time] * SVR:USPTO-EFXRF-5/19 ■ DNIS:2738300 * CSID:2 125880500 * DURATION (mm-ss):20-54 



07/16/2009 13:17 FAX 2125880500 FROMMER LAWRENCE & HAUG 12)016/051 



115 

area supplied via switch 252 and the foreground component supplied form the 
separating unit 25 1 to generate a foreground component picture. 

At step S209, the synthesis unit 314 synthesizes the background component, 
belonging to the uncovered background area, extracted by the processing of step S204, 
and the background component, belonging to the covered background area, extracted 
by the processing of step S206. The synthesized foreground component is routed to 
the synthesis unit 255, which then synthesizes the pixels belonging to the foreground 
area supplied via switch 254 and the background component supplied form the 
separating unit 251 to generate a background component picture. 

At step S2 10, the synthesis unit 253 outputs the foreground component picture. 
At step S211, the synthesis unit 255 outputs the background component picture to 
terminate the processing. 

In this manner, the foreground/background separating unit 105 is able to 
separate the foreground component and the background component from the input 
picture, based on the area information and the mixing ratio 0C, to output a foreground 
component picture, made up only of the foreground components, and the background 
component picture, made up only of the background components. 

The adjustment of the quantity of the motion blurring from the foreground 
component picture is explained. 

In Fig.56, which is a block diagram showing an illustrative structure of the 
motion blurring adjustment unit 106, the motion vector supplied from the motion 
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detection unit 102, the corresponding position information and the area information 
supplied from the area specifying unit 103 are routed to a processing unit decision unit 
351 and to a modelling unit 352. The foreground component picture supplied from the 
foreground/background separating unit 105 is sent to an addition unit 354. 

The processing unit decision unit 351 routes the generated processing unit to 
the modelling unit 352, along with the motion vector, based on the motion vector, the 
corresponding position information and the area information. 

The processing unit, generated by the processing unit decision unit 351, 
represents consecutive pixels, beginning from a pixel corresponding to the covered 
background area of the foreground component picture and extending up to a pixel 
corresponding to the uncovered background area, along the movement direction, or 
consecutive pixels, beginning from a pixel corresponding to the uncovered background 
area and extending up to a pixel corresponding to the covered background area, along 
the movement direction, as shown for example in Fig. 57. The processing unit is made 
up e.g., of an upper left point and a lower right point. The upper left point is the 
position of a pixel specified by the processing unit and lying at the leftmost or 
uppermost point on a picture. 

The modelling unit 352 executes the modelling based on the motion vector and 
on the input processing unit. More specifically, the modelling unit 352 may hold at 
the outset plural models corresponding to the number of pixels contained in the 
processing unit, the number of times of the virtual splitting of the pixel values in the 
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time axis direction and the number of the pixel-based foreground components to select 
a model specifying the correspondence between the pixel values and the foreground 
components, based on the processing unit and on the number of times of the virtual 
splitting of pixel values in the time axis direction, as shown in Fig.58. 

For example, with the number of pixels corresponding to a processing unit 
being 1 2 and with the movement quantity v in the shutter time being 5, the modelling 
unit 352 selects a sum total of eight foreground components, with the leftmost pixel 
including one foreground component, the second left pixel including two foreground 
components, the third left pixel including three foreground components, the fourth left 
pixel including four foreground components, the second left pixel including two 
foreground components, the third left pixel including three foreground components, 
the fourth left pixel including four foreground components, the fifth left pixel 
including five foreground components, the sixth left pixel including five foreground 
components, the seventh left pixel including five foreground components, the eighth 
left pixel including five foreground components, the ninth left pixel including four 
foreground components, the tenth left pixel including three foreground components, 
the eleventh left pixel including two foreground components, and the twelfth left pixel 
including one foreground component. 

Instead of selecting from the pre-stored model, the modelling unit 352 may 
generate the model based on the motion vector and on the processing unit, when the 
motion vector and the processing unit are supplied thereto. 
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The modelling unit 352 sends a selected model to an equation generating unit 

353. 

The equation generating unit 353 generates an equation based on the model 
supplied from the modelling unit 352. Referring to the model of the foreground 
component picture, shown in Fig. 5 8, an equation generated by the equation generating 
unit 353 when the number of the foreground components is 8, the number of pixels 
corresponding to the processing unit is 12, the movement quantity v is 5, and the 
number of times of the virtual splitting is 5 is explained. 

When the foreground components corresponding to the shutter time/v contained 
in the foreground component picture are FOl/v to F08/v, the relation between the 
foreground components FO 1 /v to F08/v and the pixel values CO 1 to C 1 2 is represented 
by the equations (39) to (50): 
C01 =F01/v 

—(39) 

C02 = F02/v + FOl/v 

-(40) 

C03 = F03/v + F02/v + FOl/v 

-(41) 

C04 = F04/v + F03/v + F02/v + FOl/v 

-(42) 

COS = F05/v + F04/v + F03/v + F02/v + FOl/v 
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C06 = F06/v + F05/v + F04/v + F03/v + F02/v 



C07 = F07/v + F06/v + F05/v + F04/v + F03/v 



C08 = F08/v + F07/v + F06/v + F05/v + F04/v 



C09 = F08/v + F07/v + F06/v + F05/v 



CIO = F08/v + F07/v + F06/v 



CI 1 = F08/v + F07/v 



-(43) 
-(44) 
-(45) 
-(46) 
-(47) 
-(48) 
-(49) 



C12 = F08/v 

•••(50) 

The equation generating unit 353 modifies the generated equations to generate 
equations. The equations generated by the equation generating unit 353 are indicated 
by the equations (5 1) to (62): 

C01 = 1-FOl/v + 0-F02/v + 0-F03/V + 0-F04/v + 0-F05/v + 0-F06/v + 0-F07/v + 
0-F08/v 

•••(51) 
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C02 = 1-FOl/v + l-F02/v + 0'F03/v + 0-F04/v + 0-F05/v + 0F06/v + 0-F07/v + 
0-F08/v 

-(52) 

C03 = 1-FOl/v + l-F02/v + l'F03/v + 0*F04/v + 0-F05/v + 0-F06/v + 0-F07/v + 
0-F08/v' 

-(53) 

C04 = 1-FOl/v + l-F02/v + l-F03/v + l'F04/v + 0-F05/V + 0-F06/v + 0F07/v + 
0-F08/v 

-(54) 

C05 = 1-FOl/v + l-F02/v + l-F03/v + lF04/v + l-F05/v + 0-F06/v + 0-F07/v + 
0-F08/V 

■(55) 

C06 = 0-FOl/v + l-F02/v + l-F03/v + l'F04/v + l-F05/v + l-F06/v + 0-F07/v + 
0-F08/v 

-(56) 

C07 = OFOl/v + 0-F02/v + l-F03/v + l-F04/v + l'F05/v + l-F06/v + l-F07/v + 

0- F08/v 

-(57) 

C08 = 0-FOl/v + 0F02/v + 0-F03/v + l-F04/v + l-F05/v + 0-F06/v + l-F07/v + 

1- F08/v 

-(58) 
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C09 = 0-FOl/v + 0-F02/v + 0-F03/V + 0'F04/v + l-F05/v + l'F06/v + l'F07/v + 
1-F08/V 

-(59) 

CIO = 0-FOl/v + 0-F02/V + 0-F03/V + 0-F04/v + 0-F05/v + l-F06/v + I-F07/v + 
lF08/v 

-(60) 

Cll = 0-FOl/v + 0-F02/V + 0-F03/v + 0-F04/v + 0-F05/v + O-F06/v + 0-F07/v + 
l-F08/v 

••(61) 

C12 = O-FOl/v + 0-F02/v + 0-F03/v + l*F04/v + l-F05/v + 0-F06/v + l-F07/v + 
l-F08/v 

-(62) 

The equations (51) to (62) may also be represented by the equation (63): 

08 

Cj = ^aij-Fi/v ... ™ 

1=01 v ' 

where j indicates the pixel position. In this case, j assumes one of values of 1 to 12. 
On the other hand, i denotes the position of the foreground value, and assumes one of 
values of 1 to 8. Aij has values of 0 or 1 in association with the values of i and j. 

If an error is taken into account, the equation (63) may be represented by the 
equation (64): 
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Cj=Y ai J' Filv + e J -(64) 



j=oj 



where ej is an error contained in a considered pixel Cj. 

The equation (64) may be rewritten to the equation (65): 

08 

ej= Cj-Y< ai J' Fi/v -(65). 
;=ol 

In order to apply the least square sum, a square sum of errors E is defined as 



indicated by the equation (66): 

12 

E = X e J 2 - (66). 

In order to minimize the error, it suffices if the value of the partial 
differentiation by a variable FK with respect to the error square sum EisO.Fkis found 
to satisfy the equation (67): 
SE ^ . Sej 

12 08 

= 2- £ {{Cj- X aij Fi 7 v)- {-akj I v) = 0 

,=oi. /=oi —(67). 

Since the movement quantity v in the equation (67) is constant, the equation 

(68): 

12 OS 

I akj ■ {Cj - £ aij ■ Fit v) = 0 ... (6g) 

can be derived. 
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Developing the equation (68) and shifting the term, we obtain the equation (69): 



12 



08 



12 



y=0l ;=01 y=oi 



(69). 



The equation (69) is expanded into eight equations by substituting one of 
integers of 1 to 8. The resulting eight equations can be represented by matrix by a sole 
equation termed a normal equation. 

An example of the normal equation, generated by the equation generating unit 
353 based on the minimum square method, is the following equation (70): 



54321000 


F01 


45432100 


F02 


34543210 


F03 


23454321 


F04 


12345432 


F05 


01234543 


F06 


00123454 


F07 


00012345 


F08 



= V 



12 

So 

Jl 

EC/ 

1-07 
10 

/=os 

09 
/-05 

/=04 
07 

ZCi 

06 

la 

05 

/=0I 



- (70). 

If the equation (70) is expressed as A*F — v*C, C, A and v are known, while F 
is unknown. On the other hand, A and v are known at the modelling stage, C becomes 
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known on inputting a pixel value in the adding operation. 

By calculating the foreground component by the normal equation which is 
based on the least square method, it is possible to effect scattering of errors contained 
in the pixel C. 

The equation generating unit 353 sends the so-generated normal equation to an 
addition unit 354. 

Based on the processing unit, supplied from the processing unit decision unit 
351, the addition unit 354 sets the pixel value C ? contained in the foreground 
component picture, in the matrix equation supplied from the equation generating unit 
353. The addition unit 354 sends the matrix, having set the pixel value C set therein, 
to an operating unit 355. 

The operating unit 355 calculates the foreground component freed of motion 
blurring Fi/v, by the processing which is based on the solution method, such as 
Gauss* JordAn erasure method, to find Fi corresponding to one of integers 0 to 8 of i 
as pixel values of the foreground freed of the motion blurring to output the foreground 
component picture composed of pixel values freed of motion blurring Fi to a motion 
blurring adding unit 356 and to a selection unit 357. 

Meanwhile, F01 to F08 are set to C03 to C10, in the foreground component 
picture freed of the motion blurring, shown in Fig. 59, in order to produce no changes 
in the position of the foreground component picture relative to the picture screen. An 
arbitrary position can be set. 
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The motion blurring adding unit 356 is able to adjust the quantity of the motion 
blurring by imparting the motion blurring adjustment quantity v' different from the 
movement quantity v, for example, the motion blurring adjustment quantity v' equal 
to one-half the movement quantity v, or the motion blurring adjustment quantity v' 
irrelevant to the movement quantity v, to adjust the value of the motion blurring 
quantity. For example, the motion blurring adding unit 356 divides the pixel value Fi 
of the foreground freed of the motion blurring by the motion blurring adjustment value 
v' to calculate the foreground component Fi/v' and sums the foreground components 
Fi/v' to generate a pixel value adjusted for the motion blurring quantity, as shown in 
Fig.60. For example, if the motion blurring adjustment quantity v' is 3, the pixel value 
C02 is (F01/v'), the pixel value C03 is (F01 + F02)/v, the pixel value C04 is (F01 + 
F02 + F03)/v' and the pixel value C05 is (F02 + F03 + F04)/v\ 

The motion blurring adding unit 356 sends the foreground component picture, 
adjusted for the motion blurring quantity, to the selection unit 357. 

Based on the selection signal corresponding to the user' s selection, the selection 
unit 357 selects one of the foreground component picture freed of the motion blurring, 
sent from the operating unit 355 and the foreground component picture from the 
motion blurring adding unit 356 adjusted for the motion blurring quantity to output the 
selected foreground component picture. 

The motion blurring adjustment unit 106 thus is able to adjust the motion 
blurring quantity based on the selection signals and the motion blurring adjustment 
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quantity v\ 

For example, if the number of pixels associated with the selection signal is eight 
and the movement quantity v is four, as shown in Fig.61, the motion blurring 
adjustment unit 106 is able to generate the matrix equation (71): 



"43210 " 


"foi" 


34321 


F02 


23432 


F03 


12343 


F04 


01234 


F05 



= V 



as 

i=0S 
07 

J =04 
06 

as 

2 c/ 

04 



-(71). 

The motion blurring adjustment unit 106 thus establishes a number of equations 
corresponding to the length of the processing unit to calculate the pixel value Fi 
adjusted for the motion blurring quantity. In similar manner, if the number of pixels 
contained in a processing unit is 100, the motion blurring adjustment unit 106 
generates 100 equations in association with the 100 pixels to calculate Fi. 

In Fig.62, showing another configuration of the motion blurring adjustment unit 
106, the parts or components similar to those shown in Fig.56 are indicated by the 
same reference numerals and are not explained specifically. 

A selection unit 361 routes the input motion vector and the corresponding 
position signal directly to the processing unit decision unit 351 and to the modelling 
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unit 352. Alternatively, the selection unit 361 substitutes the motion blurring 
adjustment quantity v s for the magnitude of the motion vector to route the motion 
vector, the magnitude of has been replaced by the riiotion blurring adjustment quantity 
v* and the corresponding position signal directly to the processing unit decision unit 
35 1 and to the modelling unit 352. 

By so doing, the processing unit decision units 351 to 355 of the motion 
blurring adjustment unit 106 of Fig. 62 are able to adjust the motion blurring quantity 
in association with the movement quantity v and with the motion blurring adjustment 
quantity v\ For example, if the movement quantity v is 5 and the motion blurring 
adjustment quantity v' is 3, the processing unit decision units 351 to 355 of the 
motion blurring adjustment unit 106 of Fig.62 executes the processing on the 
foreground component picture, with the movement quantity v of Fig. 58 equal to 5, in 
accordance with the model shown in Fig. 60 corresponding to the motion blurring 
adjustment quantity v' equal to 3, to calculate a picture containing the motion blurring 
corresponding to the movement quantity v of (movement quantity v)/ (motion blurring 
adjustment quantity v') = 5/3, that is approximately 1.7. Since the calculated picture 
is free of the motion blurring corresponding to the movement quantity v equal to 3, 
attention is to be directed to the fact that the relation between the movement quantity 
v and the motion blurring adjustment quantity v* has a different meaning from the 
results of the motion blurring adding unit 356. 

The motion blurring adjustment unit 106 generates an equation in association 
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with the movement quantity v and the processing unit and sets the pixel values of the 
foreground component picture in the generated equation to calculate the foreground 
component picture adjusted for the motion blurring quantity. 

Referring to the flowchart of Fig.63, the processing for adjusting the motion 
blurring quantity in the foreground component picture by the motion blurring 
adjustment unit 106 is explained. 

At step S251, the processing unit decision unit 351 of the motion blurring 
adjustment unit 106 generates a processing unit, based on the motion vector and the 
area information, to send the generated processing unit to the modelling unit 352. 

At step S252, the modelling unit 352 of the motion blurring adjustment unit 106 
selects and generates a model in association with the movement quantity v and the 
processing unit. At step S253, the equation generating unit 353 generates the normal 
equation, based on the selected model. 

At step S254, the addition unit 354 sets pixel values of the foreground 
component picture in the so-generated normal equation. At step S255, the addition 
unit 354 verifies whether or not the pixel values of the totality of pixels of the 
processing unit have been set. If it is verified that the pixel values of the totality of 
pixels corresponding to the processing unit have not been set, the program reverts to 
step S254 to repeat the processing of setting pixel values in the normal equation. 

If it is decided at step S255 that the pixel values of the totality of pixels 
corresponding to the processing unit have been set, the program reverts to step S256 
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where the operating unit 355 calculates the pixel values of the foreground, adjusted for 
the motion blurring quantity, based on the normal equation from the addition unit 354, 
in which the pixel values have been set, to terminate the processing. 

In this manner, the motion blurring adjustment unit 106 is able to adjust the 
motion blurring quantity based on the motion vector and the area information, from 
the foreground component picture containing the motion blurring. 

That is, the motion blurring adjustment unit 106 is able to adjust the motion 
blurring quantity in the pixel values as sampling data. 

Meanwhile, the structure of the motion blurring adjustment unit 106, shown in 
Fig.56, is merely exemplary and is not intended to limit the present invention. 

The signal processor 12, the configuration of which is shown in Fig. 10, is able 
to adjust the quantity of the motion blurring contained in the input picture. The signal 
processor 12, the configuration of which is shown in Fig. 10, is able to calculate the 
mixing ratio 0C as the buried information to output the so-calculated mixing ratio a. 

Fig. 64 is a block diagram showing a modified configuration of the functions of 
the signal processor 12. 

The parts or components similar to those of Fig. 10 are indicated by the same 
reference numerals and are not explained specifically. 

The area specifying unit 103 sends the area information to the mixing ratio 
calculating unit 104 and to the synthesis unit 371. 

The mixing ratio calculating unit 104 sends the area information to the 



PACE 30/51 * RCVD AT 7/16/2009 1:11 :05 PM [Eastern Daylight Time] * SVR:USPTO-EFXRF-5/19 * DNIS;2738300 " CSID:2 125880500 * DURATION (mm-SS):20-54 



07/16/2009 13:23 FAX 2125880500 FROMMER LAWRENCE & HAUG ©031/051 



130 

foreground/background separating unit 105 and to the synthesis unit 371 . 

The foreground/background separating unit 105 sends the foreground 
component picture to the synthesis unit 371. 

Based on the mixing ratio a supplied from the mixing ratio calculating unit 104 
and on the area information supplied from the area specifying unit 103, the synthesis 
unit 371 synthesizes an optional background picture and the foreground component 
picture supplied from the foreground/background separating unit 105 to output a 
picture synthesized from the optional background picture and the foreground 
component picture. 

Fig.65 shows the configuration of the synthesis unit 371. A background 
component generating unit 381 generates a background component picture, based on 
the mixing ratio a and on an optional background picture, to route the so-generated 
background component picture to a mixed area picture synthesis unit 382. 

The mixed area picture synthesis unit 382 synthesizes the background 
component picture supplied from the supplied from the background component 
generating unit 38 1 and the foreground component picture to generate a mixed area 
synthesized picture which is routed to a picture synthesis unit 383. 

Based on the area information, the picture synthesis unit 383 synthesizes the 
foreground component picture,, mixed area synthesized picture supplied from the 
mixed area picture synthesis unit 382 and an optional background picture to generate 
and output a synthesized picture. 
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In this manner, the synthesis unit 371 is able to synthesize the foreground 
component picture to an optional background picture. 

The picture obtained on synthesis of a foreground component picture with an 
optional background picture, based on the mixing ratio a, as a characteristic value, is 
more spontaneous than a picture obtained on simply synthesizing the pixels. 

Fig.66 shows, in a block diagram, a further configuration of the function of the 
signal processor 12 adapted for adjusting the motion blurring quantity. The signal 
processor 12 shown in Fig. 10 calculates the mixing ratio a and specifies the area 
sequentially, whereas the signal processor 12 shown in Fig.66 specifies the area and 
calculates the mixing ratio 0C by parallel processing. 

The functions similar to those shown in the block diagram of Fig. 10 are denoted 
by the same reference numerals and are not explained specifically. 

The input picture is sent to a mixing ratio calculating unit 401, 
foreground/background separating unit 402, an area specifying unit 103 and to an 
obj ect extraction unit 1 0 L 

Based on the input picture, the mixing ratio calculating unit 40 1 calculates, for 
each of the pixels contained in the input picture, the estimated mixing ratio in case the 
pixel is assumed to belong to the covered background area, and the estimated mixing 
ratio in case the pixel is assumed to belong to the uncovered background area, to 
su Ppty the so-calculated estimated mixing ratio in case the pixel is assumed to belong 
to the covered background area and estimated mixingratio in case the pixel is assumed 
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to belong to the uncovered background area, to the foreground/background separating 
unit 402. 

Fig. 67 shows, in a block diagram an illustrative structure of the mixing ratio 
calculating unit 40 1 . 

The estimated mixing ratio processor 201, shown in Fig.67, is similar to the 
estimated mixing ratio processor 201 shown in Fig.37. The estimated mixing ratio 
processing unit 202, shown in Fig.67, is the same as the estimated mixing ratio 
processing unit 202 shown in Fig.37. 

The estimated mixing ratio processor 20 1 calculates the estimated mixing ratio, 
from pixel to pixel, by calculations corresponding to the model of the covered 
background area, based on the input picture, to output the so-calculated estimated 
mixing ratio. 

The estimated mixing ratio processor 202 calculates the estimated mixing ratio, 
from pixel to pixel, by calculations corresponding to the model of the uncovered 
background area, based on the input picture, to output the so-calculated estimated 
mixing ratio. 

Based on the estimated mixing ratio in case the pixel is assumed to belong to 
the covered background area, and the estimated mixing ratio in case the pixel is 
assumed to belong to the uncovered background area* supplied from the mixing ratio 
calculating unit 401, and on the area information, supplied from the area specifying 
unit 103, the foreground/background separating unit 402 generates a foreground 
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component picture from the input picture, to route the so-generated foreground 
component picture to the motion blurring adjustment unit 106 and to the selection unit 
107. 

Fig. 6 8 is a block diagram showing an illustrative structure of the 
foreground/background separating unit 402. 

The parts or components similar to those of the foreground/background 
separating unit 105 shown in Fig.48 are indicated by the same reference numerals and 
not explained specifically. 

Based on the area information supplied from the area specifying unit 103, a 
selection unit 42 1 selects one of the estimated mixing ratio in case the pixel is assumed 
to belong to the covered background area, and the estimated mixing ratio in case the 
pixel is assumed to belong to the uncovered background area, supplied from the 
mixing ratio calculating unit 40 1 , and routes the so-selected estimated mixing ratio as 
the mixing ratio (X to the separating unit 25 1. 

Based on the mixing ratio a and the area information, supplied from the 
selection unit 42 1, the separating unit 251 separates the foreground components and 
the background components from the pixel values of pixels belonging to the mixed 
area, to send the foreground components extracted to the synthesis unit 253, as well 
as to send the background components to the synthesis unit 255. 

The separating unit 251 may be configured similarly to the structure shown in 
Fig.53. 
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The synthesis unit 253 synthesizes and outputs the foreground component 
picture. The synthesis unit 255 synthesizes and outputs the background component 
picture. 

The motion blurring adjustment unit 106, shown in Fig.66, may be configured 
as in Fig. 1 0 . Based on the area infonnation and the motion vector, the motion blurring 
adjustment unit 106 adjusts the quantity of the motion blurring supplied from the 
foreground/background separating unit 402, to output the foreground component 
picture adjusted for the motion blurring quantity. 

Based on the selection signal, corresponding to the selection by the user, the 
selection unit 107 selects one of the foreground component picture supplied from the 
foreground/background separating unit 402 and the foreground component picture 
from the motion blurring adjustment unit 106, adjusted for the motion blurring 
quantity, to output the selected foreground component picture. 

In this manner, the signal processor 12, the configuration of which is shown in 
Fig.66, is able to adjust a picture, corresponding to an object of the foreground object 
contained in the input picture, to output the resulting picture. The signal processor 12, 
the configuration of which is shown in Fig.66, is able to calculate the mixing ratio a, 
as the buried information, as in the first embodiment, to output the so-calculated 
mixing ratio Ot. 

Fig. 69 is a block diagram showing a modification of the function of the signal 
processor 1 2 adapted for synthesizing the foreground component picture to an optional 
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background picture. The signal processor 12, shown in Fig.64, performs area 
identification and the calculation of the mixing ratio a in series, whereas the signal 
processor 12, shown in Fig.69, performs area identification and the calculation of the 
mixing ratio CC in parallel. 

The functions similar to those shown in the block diagram of Fig.66 are denoted 
by the same reference numerals and are not explained specifically. 

Based on the input picture, the mixing ratio calculating unit 401, shown in 
Fig.69, calculates the estimated mixing ratio for when the pixel is assumed to belong 
to the covered background area and the estimated mixing ratio for when the pixel is 
assumed to belong to the uncovered background area, for each of the pixels contained 
in the input picture, to route the estimated mixing ratio for when the pixel is assumed 
to belong to the covered background area and the estimated mixing ratio for when the 
pixel is assumed to belong to the uncovered background area, to the 
foreground/background separating unit 402 and to the synthesis unit 43 1 . 

Based on the estimated mixing ratio for when the pixel is assumed to belong to 
the covered background area, the estimated mixing ratio for when the pixel is assumed 
to b elong to the uncovered background area, supplied from the mixing ratio calculating 
unit 401, and on the area information supplied from the area specifying unit 103, the 
foreground/background separating unit 402, shown in Fig. 69, generates the foreground 
component picture from the input picture to route the generated foreground component 
picture to the synthesis unit 431. 
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Based on the estimated mixing ratio for when the pixel is assumed to belong to 
the covered background area, the estimated mixing ratio for when the pixel is assumed 
to belong to the uncovered background area, supplied from the mixing ratio calculating 
unit 401, and on the area information supplied from the area specifying unit 103, the 
synthesis unit 431 synthesizes an optional background area and a foreground 
component picture supplied from the foreground/background separating unit 402, to 
output a picture synthesized from the optional background area and the foreground 
component picture. 

Fig. 70 shows the configuration of the synthesis unit 43 1 . The functions similar 
to those shown in the block diagram of Fig.65 are denoted by the same reference 
numerals and are not explained specifically. 

Based on the area information, supplied from the area specifying unit 103, a 
selection unit 441 selects one of the estimated mixing ratio for when the pixel is 
assumed to belong to the covered background area and the estimated mixing ratio for 
when the pixel is assumed to belong to the uncovered background area, supplied from 
the mixing ratio calculating unit 401, to route the selected estimated mixing ratio as 
the mixing ratio a to the background component generating unit 381 . 

Based on the mixing ratio 0C supplied from the selection unit 441 and the 
optional background component picture, the background component generating unit 
381, shown in Fig. 70, generates a background component picture, to route the 
generated picture to the mixed area picture synthesis unit 382. 
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The mixed area picture synthesis unit 382, shown in Fig.70, synthesizes the 
background component picture, supplied from the background component generating 
unit 381, to the foreground component picture, to generate a mixed area synthesized 
picture, which is routed to the picture synthesis unit 383. 

Based on the area information, the picture synthesis unit 383 synthesizes 
foreground component picture, the mixed area synthesized picture, supplied from the 
mixed area picture synthesis unit 382 and an optional background picture, to generate 
and output a synthesized picture. 

In this manner, the synthesis unit 431 is able to synthesize the foreground 
component picture to an optional background picture. 

Although the mixing ratio a has been explained as a proportion of the 
background component contained in the pixel value, it may also be a proportion of the 
foreground component contained in the pixel value. 

Although the direction of movement of the object as the foreground has been 
explained as being from left to right, this direction is, of course, not limitative. 

An embodiment in which the amount of the motion blurring quantity contained 
in temperature or pressure data by the similar processing as that performed by the 
signal processor 12 is explained. 

Fig.71 shows an illustrative structure of a signal processing apparatus according 
to the present invention. A thermography device 451 detects IRrays, radiated from an 
object being measured from an enclosed IR sensor, such as an IR CCD, to generate a 
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signal corresponding to the wavelength or intensity of the detected IR rays. The 
thermography device 451 analog/digital converts the generated signal to compare the 
converted signal to reference data corresponding to the reference temperature to 
generate temperature data indicating the temperature of various sites of the object to 
output the generated temperature data to the signal processor 452. 

Similarly to the sensor 1 1, the thermography device 45 1 has integrating effects 
with respect to the space and time. 

The temperature data the thermography device 451 routes to the signal 
processor 452 is configured similarly to the picture data of the moving picture, and is 
such data in which the values indicating the temperature of respective sites of the 
object being measured (corresponding to the pixel values of the picture data) are 
arrayed two-dimensionally along the spatial direction in association with the picture 
data frames and also are arrayed along the temporal direction. 

The signal processor 452 adjusts the distortion contained in the input 
temperature data and which has been generated as a result of movement of the object 
being measured. For example, the signal processor 452 extracts a more accurate 
temperature of the desired site of the object being measured. 

Fig. 72 is a flowchart showing the processing for adjusting the motion blurring 
quantity by the signal processor 452. At step S30 1 , the signal processor 452 acquires 
temperature data in which values indicating the temperatures for respective sites of the 
object being measured are arrayed two-dimensionally. Based on the temperature data, 
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the signal processor 452 generates data specifying the movement. 

At step S302, the signal processor 452 specifies areas of temperature data to a 
foreground area comprising only the values indicating the temperature corresponding 
to a desired object, a background area comprising only the values indicating the 
temperature corresponding to an object other than the desired object, and a mixed area 
comprising the temperature information corresponding to the desired object and the 
temperature information corresponding to the object other than the desired object. 

At step-S303, the signal processor 452 checks whether or not the temperature 
indicating value belongs to the temperature data. If the signal processor 452 decides 
that the temperature indicating value belongs to the mixed area, the signal processor 
452 proceeds to step S3 04 to calculate the mixing ratio 0C by the processing similar to 
that of step S 102 of Fig.27. 

At step S305, the signal processor 452 separates the information of the 
temperature corresponding to the object desiring temperature measurement, by the 
processing similar to the processing of step S103 of Fig.27, to then proceed to step 
S306. 

For separating the temperature information at step S305, the temperature 
information may be converted, based on the KirchhofF's law or the law specifying the 
relation between the object temperature and the radiated IR rays, such as Stephen- 
Boltzmann law, into the energy quantity of the IR rays, emitted from the object 
desiring temperature measurement, to separate the energy quantity of the converted 
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IR rays to re-convert the separated energy quantity into temperature. By conversion 
into the IR ray energy prior to separation, the signal processor 452 is able to separate 
the temperature information more accurately than in direct separation of the 
temperature information. 

If, at step S303, the temperature indicating value contained in the temperature 
data does not belong to the mixed area, it is unnecessary to separate the temperature 
information corresponding to the object desiring the temperature measurement. So, 
the processing at steps S304 and S305 are skipped so that the processing proceeds to 
step S306. 

At step S306, the signal processor 452 generates temperature data for causing 
temperature measurement to correspond to the desired object, from a value indicating 
the temperature belonging to the foreground temperature and the information on the 
temperature which causes the temperature measurement to correspond to the desired 
object. 

At step S307, the signal processor 452 generates a model corresponding to the 
generated temperature data by the processing similar to the processing at step S251. 

At step S308, the signal processor 452 adjusts the quantity of the motion 
blurring contained in the temperature data corresponding to the object, in need of 
temperature measurement, by the processing similar to that of steps S252 to S255 of 
Fig.63, based on the generated model, to terminate the processing. 

In this manner, the signal processor 452 adjusts the quantity of the motion 
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blurring contained in the temperature data generated by the movement of the object 
being measured to calculate the more accurate temperature difference of respective 
object portions. 

Fig.73 shows an illustrative structure of a signal processing apparatus according 
to the present invention for weighing measurement. The pressure area sensor 50 1 is 
made up of plural pressure sensors to measure the load per unit planar area, that is the 
pressure. . The pressure area sensor 501 is of a structure comprised of a two- 
dimensional array on a floor surface of plural pressure sensors 511-1-1 to 51 1-M-N. 
When an object 512, the weight of which is being measured, is moved on the pressure 
area sensor 501, the pressure area sensor 501 measures the pressure applied to each 
of the pressure sensors 511-1-1 to 51 1-M-N to generate weight data for each of 
measured ranges of the pressure sensors 511-1- 1' to 51 1-M-N to output the generated 
weight data to the signal processor 502. 

The pressure sensors 511-1-1 to 51 1-M-N are each made up of a sensor 
exploiting double refraction produced when an external force is applied to a 
transparent elastic material, or the so-called photoelasticity. 

The pressure area sensor 501 in its entirety may be constructed by a sensor 
exploiting the photoelasticity. 

Fig. 75 illustrates the load associated with the weight of each part of the object 
512 applied to respective ones of the pressure sensors 511-1-1 to 51 1-M-N making up 
the pressure area sensor 501. 
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The load A corresponding to the weight of the leftmost portion of the object 5 1 2 
in Fig.75 is applied to the pressure sensor 5 1 1-m-l. The load b corresponding to the 
weight of the second left portion of the object 512 is applied to the pressure sensor 
51 l-m-2. The load c corresponding to the weight of the fourth left portion of the 
object 5 12 is applied to the pressure sensor 5 1 l-m-3. The load d corresponding to the 
weight of the fourth left portion of the object 5 1 2 is applied to the pressure sensor 511- 
in-4. 

The load e corresponding to the weight of the second left portion of the object 
5 12 is applied to the pressure sensor 5 1 l-m-5. The load f corresponding to the weight 
of the fourth left portion of the object 5 12 is applied to the pressure sensor 5 1 l-m-6. 
The load g corresponding to the weight of the fourth left portion of the object 5 12 is 
applied to the pressure sensor 51 l-m-7. 

The weight data output by the pressure area sensor 501 corresponds to the 
arrangement of the pressure sensor 511-1-1 to 511-M-N and is comprised of weight 
values arrayed two-dimensionally in the spatial direction. 

Fig.76 illustrates typical weight data output by the pressure area sensor 501 
when the object 512 is moving, with the pressure area sensor 501 having integrating 
effects. 

On the pressure sensor 5 1 1-m- 1 is applied a load A corresponding to the weight 
of the leftmost portion of the object 5 12 in a unit time of measurement, a value A is 
output as a value indicating the weight included in the weight data. 
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In the unit time for measurement, there are applied to the pressure sensor 511- 
m-2 a load b corresponding to the weight of the second left portion of the object 512, 
and a load d corresponding to the weight of the leftmost portion of the object 5 12,so 
the pressure sensor 511-m-2 outputs a value A+b as a value indicating the weight 
comprehended in the weight data: 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-3 a load c corresponding to the weight of the third left portion of the object 5 12, a 
load b corresponding to the weight of the second left portion of the object 512 and 
subsequently the load A corresponding to the weight of the second left portion of the 
object 512, so the pressure sensor 511-m-2 outputs a value A+b+c as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-4 a load d corresponding to the weight of the fourth left portion of the object 512, 
a load c corresponding to the weight of the third left portion of the object 5 12, a load 
b corresponding to the weight of the second left portion of the object 512 and 
subsequently the load A corresponding to the weight of the leftmost portion of the 
object 512, so the pressure sensor 511-m-2 outputs a value A+b+c+d as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-5 a load e corresponding to the weight of the fifth left portion of the object 5 12, a 
load d corresponding to the weight of the fourth left portion of the object 5 12, a load 
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c corresponding to the weight of the third left portion of the object 512 and 
subsequently the load A corresponding to the weight of the second left portion of the 
object 512, so the pressure sensor 511-m-5 outputs a value b+c+d+e as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-6 a load f corresponding to the weight of the sixth left portion of the object 512, a 
load e corresponding to the weight of the fifth left portion of the object 5 12, a load d 
corresponding to the weight of the fourth left portion of the object 512 and 
subsequently the load c corresponding to the weight of the third left portion of the 
object 512, so the pressure sensor 511-m-6 outputs a value c+d+e+f as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-7 a load g corresponding to the weight of the seventh left portion of the object 5 12, 
a load f corresponding to the weight of the sixth left portion of the object 5 12, a load 
e corresponding to the weight of the fifth left portion of the object 512 and 
subsequently the load d corresponding to the weight of the fourth left portion of the 
object 512, so the pressure sensor 511-m-7 outputs a value d+e+ffg as a value 
indicating the weight comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-8 a load g corresponding to the weight of the seventh left portion of the object 512, 
a load f corresponding to the weight of the sixth left portion of the object 512, and a 
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load e corresponding to the weight of the fifth left portion of the object 5 12, so the 
pressure sensor 511-m-8 outputs a value e+f+-g as a value indicating the weight 
comprehended in the weight data. 

In the unit time for measurement, there are applied to the pressure sensor 511- 
m-9 a load g corresponding to the weight of the seventh left portion of the object 512, 
and a load f corresponding to the weight of the sixth left portion of the object 512, so 
the pressure sensor 511-m-9 outputs a value ffg as a value indicating the weight 
comprehended in the weight data. 

In the unit time for measurement, there is applied to the pressure sensor 511 -m- 
10 a load g corresponding to the weight of the seventh left portion of the object 512 , 
so the pressure sensor 511-m-10 outputs a value g as a value indicating the weight 
comprehended in the weight data. 

The pressure area sensor 501 outputs weight data comprised of the value A 
output by the pressure sensor 51 1-m-l , the value A+b output by the pressure sensor 
511 -m-2 , the value A+b+c output by the pressure sensor 511 -m-3, the value A+b+c+d 
output by the pressure sensor 51 l-m-4, the value b+c+d+e output by the pressure 
sensor 51 l-m-4, the value A+b+c output by the pressure sensor 51 l-m-3, the value 
A+b+c+d output by the pressure sensor 51 l-m-4, the value b+c+d+e output by the 
pressure sensor 51 l-m-5, the value c+d+e+f output by the pressure sensor 51 l-m-6, 
the value d+e+ffg output by the pressure sensor 511 -m-7, the value e+f+g output by 
the pressure sensor 51 l-m-8, the value f^-g output by the pressure sensor 51 l-m-9, 
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and the value g output by the pressure sensor 5 1 l-m-10. 

The signal processor 502 adjusts the distortion generated by the movement of 
the object 512 being measured from the weight data supplied from the pressure area 
sensor 501. For example, the signal processor 502 extracts more accurate weight of 
the desired sites of the object 512 being measured. For example, the signal processor 
502 extracts the loads A and b to g from weight data comprised of the value A, A+b, 
A+b+c, A+b+c+ d, b+c+d+e, c-hd+e+f, d+e+f+g, e+f^g, f^g and g. 

Referring to the flowchart of Fig.77, the processing for calculating the load 
executed by the signal processor 502 is explained. 

At step S401, the signal processor 502 acquires weight data output by the 
pressure area sensor 501. At step S402 , the signal processor 502 decides, based on the 
weight data acquired from the pressure area sensor 501, whether or not the load of the 
object 512 is being applied to the pressure area sensor 501. If it is decided that the 
load of the object 512 is being applied to the pressure area sensor 501, the signal 
processor 502 proceeds to step S403 to acquire the movement of the object 5 12 based 
on changes in the weight data. 

At step S404, the signal processor 502 acquires one-line data of the pressure 
sensor 511 contained in the weight data along the direction of movement acquired by 
the processing at step S403. 

At step S405, the signal processor 502 calculates the load corresponding to the 
weight of the respective portions o the object 5 12 to terminate the processing. The 
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signal processor 502 calculates the load corresponding to the weights of respective 
parts of the object 512 by a processing similar to the processing explained with 
reference to the flowchart of Fig.63. 

If, at step S402, the load of the object 512 is not applied to the pressure area 
sensor 501, there is no weight data to be processed, so the processing is terminated. 

In this manner, the weight measurement system is able to calculate the correct 
load corresponding to the weight of each portion of the moving object. 

The signal processor 12 generating a picture of higher resolution in the spatial 
direction is explained. 

Fig.78 is a block diagram showing a configuration of generating high resolution 
picture by increasing the number of pixels per frame, as another function of the signal 
processor 12. 

A frame memory 701 stores an input picture on the frame basis and sends the 
stored picture to a pixel value generator 702 and to a correlation calculating unit 703. 

The correlation calculating unit 703 calculates the correlation values of pixel 
values neighboring to one another in a transverse direction, contained in a picture 
supplied from the frame memory 70 1, to send the calculated correlation values to the 
pixel value generator 702. The pixel value generator 702 calculates double density 
picture components from the pixel values of the center pixel, based on the correlation 
values supplied from the correlation calculating unit 703, to generate a horizontal 
double-density picture, with the so-calculated picture component as the pixel value. 
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The pixel value generator 702 sends the so-generated horizontal double-density picture 
to the frame memory 704. 

The frame memoiy 704 stores the horizontal double-density picture, supplied 
from the pixel value generator 702, to send the so-stored horizontal double-density 
picture to a pixel value generating unit 705 and to a correlation calculating unit 706. 

The correlation calculating unit 706 calculates the correlation values of pixel 
values neighboring to one another in a vertical direction, contained in a picture 
supplied from the frame memory 704, to send the calculated correlation values to the 
pixel value generator 705. The pixel value generator 705 calculates double density 
picture components from the pixel values of the center pixel, based on the correlation 
values supplied from the correlation calculating unit 703, to generate a horizontal 
double-density picture, with the so-calculated picture component as the pixel value. 
The pixel value generator 705 outputs the so-generated horizontal double-density 
picture. 

The processing for generating a horizontal double-density picture by the pixel 
value generator 702 is explained. 

Fig.79 shows an arrangement of pixels provided in a sensor 1 1 as a CCD, and 
an area for pixel data of a horizontal double-density picture. In Fig.79, A to I indicate 
individual pixels. The areas A to r each denote a light reception area obtained on 
halving the individual pixels A to I in the longitudinal direction. With the width 21 of 
the light reception area of each of the pixels A to I, the width of each of the areas A 
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to r is I. The pixel value generator 702 calculates the pixel values of the pixel data 
associated with the areas A to r. 

Fig. 80 illustrates pixel data corresponding to the light incident on the areas A 
to r. In Fig. 80, f(x) denotes an spatially ideal pixel value in association with the input 
light and to the spatially tiny domain. 

If a pixel value of a pixel data is represented by the uniform integration of the 
ideal pixel value f (x)„ the pixel value Yl of pixel data associated with the area i is 
represented by the equation (72): 

n =/<*>&. i ... (72) 

whilst the pixel value Y2 of the picture data associated with the area j and the pixel 
value Y3 of the pixel E are represented by the following equations (73) and (74): 



f' 3 1 
72 = J f{x)dx- 



>*i' — e -(73) 
and 

= ^ - (74) 

respectively. 

In the above equations (72) to (74), xl s x2 and x3 are spatial coordinates of the 
respective boundaries of the light reception area, area i and the area j of the pixel E, 
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respectively. 

By modifying the equation (74), the following equations (75), (76) may be 
derived: 

Yl =2-Y3-Y2 

^(75) 

and 

Y2 - 2-Y3-Y1 

-(76). 

Therefore, if the pixel value Y3 of the pixel E and the pixel value Y2 of the 
pixel data corresponding to the area j are known, the pixel value Yl of the pixel data 
corresponding to the area i may be calculated from the equation (75). Also, if the pixel 
value Y3 of the pixel E and the pixel value Yl of the pixel data corresponding to the 
area i are known, the pixel value Y2 of the pixel data corresponding to the area j can 
be calculated from the area j. 

If the pixel value corresponding to a pixel and one of pixel values of the pixel 
data corresponding to the two areas of the pixel are known, the pixel value of the other 
pixel data corresponding to the other areas of the pixel may be calculated. 

Referring to Fig.8 1 , the manner of calculating the pixel values of the pixel data 
corresponding to the two areas of one pixel is explained. Fig.8 1 A shows the relation 
between the pixels D, E and F and the spatially ideal pixel value f (x). 

Since the pixels D to F own integrating effects, and one pixel outputs a pixel 
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